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Abstract. We characterise and enumerate permutations that are sortable by 
n — 4 passes through a stack. We conjecture the number of permutations 
sortable by n — 5 passes, and also the form of a formula for the general case 
n — k, which involves a polynomial expression. 



1. Background 

We view permutations as words without repeated letters; if tt is a permutation 
of an n element set and 7r(i) = tt^, then we write tt = tti . . . 7r„. The stack sorting 
operator S can be defined recursively on permutations of finite subsets of {1, 2, ... } 
as follows. If TT is empty then S{7r) = tt. If tt is nonempty write tt as the concatena- 
tion TT — LnR, where n is the greatest element of tt and L and R are the subwords 
to the left and right of n respectively. Then 

S{tt) = S{L)S{R)n. 

For example, S'(42513) — 24135. We say that a permutation tt is k-stack sortable if 
S''{tt) = id, where S'^' ^ S o 5" is the identity operator and id is the identity 

permutation 12 ... n. Let the (stack sorting) complexity of vr, denoted ssctt, be the 
smallest k such that tt is fc-stack sortable. Let 6„ be the set of permutations of 
{1, . . . , n}. For the permutations in 63 we have 



TT = 123 


132 


213 


231 


312 


321 


SSC TT — 


1 


1 


2 


1 


1 



Let Wn,k be the set of all fc-stack sortable permutations in (5„; in other words, 
Wn,k is the set of permutations in 6„ whose complexity is at most fc. Let E„^k be 
the set of permutations in 6„ whose complexity is exactly k. Note that 

En,k = Wn,k — Wn,k-1', 

Wn,k = &n — {En,k+1 U • • • U En^n-l). 

It is easy to see that Wn,n-i = 6„. Knuth ^ 2.2.1.5] leaves as an exercise to 
the reader to show that Wn,i — 6n(231). In his PhD thesis West [3] showed that 

W^n,2 = 6^(2341,35241), 

where S„(2341, 35241) is the set of permutations in (3„ that avoid the pattern 
2341 and the "barred" pattern 35241; for information on these, see [2], especially 
Section 7. West also showed that Wn,n-2 are precisely those permutations that 
do not have sufRx nl. This last statement is easily shown by proving that the 
permutations in En^n-i are those with suffix nl. 

In addition. West characterized En_n-2- To state that result it is convenient to 
introduce some notation for special sets of words over the alphabet {1, 2, . . . }. Let 
an asterisk (*) stand for any word of zero or more characters, and let a question 
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mark (?) stand for any single letter. These conventions are adopted from the so 
called glob patterns in computer science. For a word w over {*, ?} U {1, 2, . . . }, let 
( w ) denote the set of words of the form w. For instance, ( *nl ) consists of all words 
with suffix nl, and 65 n (*51?) = {23514,24513,32514,34512,42513,43512}. Let 

(wi, . . . , Wfc ) = ( wi ) U • • • U (wfc ). 

West's characterisation of En^n-i and can then be stated as in the following 

two lemmas, which follow from Theorems 4.2.4 and 4.2.17 in [3] and their proofs. 

Lemma 1.1. For all n > 2, 

En.n-l = 6n n (*nl). 

Thus, the cardinality of E„^n~-i is {n — 2)!. 
Lemma 1.2. For all n> A, 

En,n-2 — S„ n ( *n2, *(n — l)ln, *nli, *n?l, *nH<(n — 2)*(n — 1)1 ). 

Thus, the cardinality of En,n-2 is {n — 3)!(7n — 12)/2. 

By subtracting the cardinalities in Lemmas 11.11 and 11.21 from n!, we get the 
following result. 

Proposition 1.3. For all n > 4 the cardinality ofWn,n-3 is 

(2n3 - 6n2 _ 5n + 16) . 

2. Permutations requiring exactly (n — 3)-stack sorts 

Theorem 2.1. For all n > 6, the set of permutations i?„.„_3 are those given in 
column labeled "Type" of Table{^ The number of such permutations is 

(n-4)!^4,/n 6\ /n 6\ (2.1) 



3 \ \ 2 J V 1 _ 

To prove this we require some terminology and results from West [3i §4.2]. A 
forbidden pattern of order fc in a permutation tt is a triple {B,c,a), where i3 is a 
subsequence of length fc in tt and (c, a) is a pair of entries in tt such that for every 
b G B the subsequence bca is an occurrence of the pattern 231, that is, a < b < c. 
In such a situation we say that the pair (c, a) witnesses the forbidden pattern B. 
We call a forbidden pattern {B, c, a) uninterrupted if there is no subsequence bxb' 
in TT where b,b' G B and x > c. 

Lemma 2.2. [3] Theorems 4.2.10 and 4.2.14] Let tt be a permutation. 

(i) ssc(7r) < k if TT does not contain any forbidden pattern of order k; 

(ii) ssc(7r) > k if TT contains an uninterrupted forbidden pattern of order k. 

Proof of Theorem \2.1\ Let tt G S„. Using the contrapositive of (i) and (ii) in 
Lemma [2.21 if ssc(7r) = n — 3, then tt contains a forbidden pattern of order ?i — 4 
and does not contain an uninterrupted forbidden pattern of order 71 — 3. The 
forbidden pattern of order n — 4 must be witnessed by entries that appear in some 
two of the positions n — 3, n ~ 2, n — 1 ornof the permutation. We will first 
condition on the position of the largest entry in the permutations, then condition 
on those permutations that contain a forbidden pattern of the required order, and 
finally single out those permutations that are in En^n-s- 

Suppose TTn = n. A permutation tt = tt'u is a member of En.n-3 precisely when 
tt' is a member of i?„_i,„_3. Thus entries l(a)-l(e) of Table [1] immediately follow 
from Lemma ll.2l 
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Case 


iype 


Number 


7r„ = 


n 




1(a) 


*(n — l)2n 


(n-3)! 


1(b) 


*(n — l)?ln 


(n-3)! 


1(c) 


*(n — l)l?n 


(n-3)! 


1(d) 


*(n — l)*(n — 3)*(n — 2)ln 


(n-3)!/2 


1(e) 


*{n — 2)l(n — l)n 


(n-4)! 



TTn-l = n 



2(a) 


*n3 


(n 


-2)! 




2(b) 


*(n — l)ln? 


(n 


-5)(n 


-4)! 


2(c) 


*(n- 2)ln(n- 1) 


(n 


-4)! 






= n 








3(a) 


*n2? 


(n 


-3)(n 


-3)! 


3(b) 


*n?2 


(n 


-3)(n 


-3)! 




= n 








4(a) 


*n??l, but not *n(n — 2)(n — 1)1 


(n 


-2)!- 


(n-4)! 


4(b) 


*n?l? 


(n 


-2)! 




4(c) 


=i!nl?? 


(n 


-2)! 




4(d) 


*n(n - 2)(n - 1)2 


(n 


-4)! 





n and i > 3 



5(a) 
5(b) 
5(c) 

5(d) 

5(e) 

5(f) 
5(g) 
5(h) 



*nA{n - 2)B{n - 1)2, where A\JB^' 
*n*(n — 3)*(n — l)(n — 2)1 
*n*(n — l)*(n — 3)*(n — 2)1 



/ n /("'-3)*(n-4); , 
*(n — l)*n*< ; ; Jsf*[n 

^ ' \(n - 4)*(n - 3) ' ^ 

*n*(n — 2)*(n — 1)< 

*n*(n — 3)*(n — l)l(n — 2) 

*n*(n — 3)*(n — 2)l(n — 1) 

/ o^ /(n-3)*(n-4)\ , 
*(n — 2)*n*<; ; J\{*\n 

^ ' 1 (n - 4)*(n - 3)J ^ 



2)1 



-1)1 



(n 


-2)!/2- 


- (n - 4)! 


(n 


- 3)!/2 




(n 


- 2)!/6 




(n 


-2)!/12 




(n 


-4)(n- 


3)! 


(n 


-3)!/2 




(n 


-3)!/2 




(n 


-2)!/12 





Table 1 . Permutations in S„ „_3 
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Suppose 7r„_i = n. We must have 7r„ > 3, for otherwise, by Lemmas 11.11 and [ 1.21 
we would have tt g En^n-i or tt £ £'„_„_2. If 7r„ = 3 then the permutation is in 
En,n-3, and hence we get 2(a). If fc = 7r„ > 3 then it is impossible for {n,k) 
to witness a forbidden pattern F of order n — 4 since there could be at most 
n — fc — l<n — 4 elements in F. Thus the forbidden pattern F must be witnessed 
by (7r„_3, 7r„_2)- As there are now n — 2 values from which to form the forbidden 
pattern F, we are forced to choose the extreme values from this set as the values 
for (7r„_3, 7r„_2). Consequently, if fc = n — 1 then we must have (7r„_3, 7r„_2) = 
(n — 2, 1), giving case 2(c). Otherwise 3 < fc < n — 1 and (7r„_3, 7r„_2) ~ {n — 1, 1) 
which gives case 2(b). One easily verifies that all such permutations are in En^n-3- 

Suppose 7r„_2 = n. By Lemma 11.21 the value 1 cannot be to the right of n. Let fc 
be the smaller of the two values 7r„_i and 7r„. If fc > 2 then the pair (n, fc) witnesses 
a forbidden pattern of order at most n — k — 1 — I < n — 4. For the same reason, 
[iTn-i, 7r„) cannot witness a forbidden pattern of order n — 4. So exactly one of the 
entries to the right of n must be 2, giving 3(a) and (b). One easily verifies that all 
such permutations are in 

Suppose 7r„_3 = n. If 1 is to the right of n in tt then (n, 1) witnesses a forbidden 
pattern of order n — 4. However, if tt G ( m{n — 2)(n — 1)1 ), it has one of the forms 
given in Lemma 11.21 In all other cases the permutation is in This gives 

4(a), (b) and (c). 

Alternatively, if 1 is to the left of n, let fc be the value of the smallest entry to 
the right of n. Since there are at most n — 5 entries to the left of rt in tt that take 
values between fc and n, the pair (n, fc) cannot witness a forbidden pattern of order 
n — 4. In order for tt to contain a forbidden pattern F of order n — 4 it must be 
witnessed by the pair (7r„_i,7r„) and have as the block of n — 4 values the value 
7r„„2 along with the n — 5 elements to the left of n that are not 1. Thus 7r„ = 2, 
7r„_i = n — 1 and 

TT G ( *nfc(n - 1)2 ) 

for some fc ^ 1. One easily checks that the only value of fc for which tt G En,n~3 is 
k — n — 2. This gives 4(d). 

Suppose TTn-i = n for some i > 3. Then (n, fc) can never witness a forbidden 
pattern of order n — 4. In the remainder of the proof, we condition on the relative 
positions of (n-2), (n-1) and n. IItt = ^nB(n-l)C(n-2)i:) where \BiJCiJD\ > 2, 
then 

S{tt) = S{A)S{B)S{C)S{D){n - 2){n ~ l)n. 

This gives 

Tl G S„^„_3 ^ S{A)S{B)S{C)S{D) G £;„-3,n-4 

^ S{A)S{B)S{C)S{D) {*{n-2,)\). 

Since jS U C U Z3| > 2, the ways in which this can happen are restricted to 

(i) D — \ and n — 3 G C, giving case 5(c), 

(ii) D — \, C — % and n — 3 G S, giving case 5(b), 

(iii) D — ^, C ^ 1 and n — 3 G B, giving case 5(f). 

If TT = AnB{n - 2)C{n - 1)D where \BUCU D\>2, then 

^(Tr) = S{A)S{B)S{C){n - 2)S(D){n - l)n. 

This implies that tt belongs to E^.n-z if and only if 

S{A)S{B)S{C){n - 2)S{D) G £;„-2,„-4. 

If £> = then TT belongs to £;„^„_3 if and only if S{A)S{B)S{C) belongs to 
En-z^n-ii which happens if and only if C = 1 and {n — 3) G -B, from which 
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we get 5(g). Otherwise S{A)S{B)S{C){n - 2)S{D) belongs to the types listed in 
Lemma FOl There are 3 cases to consider: 

(i) S{D) = 2, so TT = AnB{n - 2)C(n - 1)2 where B U C ^ 0, giving case 5(a); 

(ii) S{D) — Ik, so D = Ik 01 D = kl, giving case 5(e); and 

(iii) it is not possible that S{A)S{B)S{C){n — 2)S{D) matches the last type in 
Lemma [TT2] since this would mean that S{D) ends in 1, its smallest entry. 

If TT = A{n — l)BnC{n — 2)D where \C U D\ > 3, then a forbidden pattern 
of order n — 4 can only be witnessed by the pair (n — 2,1). Furthermore, all 
elements 2 through n — 3 must be to the left of n — 2, so 7r„_i = n — 2 and 
7r„ = 1, that is TT = A{n — l)BnC{n — 2)1 where \C\ > 2. From this we have 
S^iw) = S{S{A)S{B))S{S{C)l){n - 2){n - l)n and 

TT e S„,„_3 ^ tt' = S{S{A)S{B))S{S{C)l) e E^^^.n-s- 

The conditions on tt' (and therefore tt) are easily derived by comparing tt' to the 
types in Lemma FOl Since \C\ > 2 one cannot have (n — 3)2 as a suffix of S{S{C)1). 
Similarly, {n — 3)lk and (n — 3)fcl cannot be suffixes of S{S{C)1). Also, it is only 
possible that 

tt' e ( *{n - 3)*(n - 5)*{n - 4)1 ) 

if C = 0, which is not allowed. The only possibility for tt' is that it has (n— 4)l(n— 3) 
as a suffix, and son — 4, n — 3gC since \C\ > 2. Under these conditions, ^^(Tr) has 
suffix (n-4)l(n-3)(n-2)(n-l)7iand tt e En^n-s- Thus D = 1 and n-4,n-3 £ C, 
from which we get case 5(d). 

If TT = A{n - 2)BnC{n - l)D where |C U > 3 then 

S{tt) = S{A)S{B){n - 2)S{C)S{D){n - l)n. 

Now, TT belongs to En,n-3 if and only if tt' = S{A)S{B){n - 2)S{C)S{D) belongs 
to -E„-2,n-4- Since |C U Dj is at least 3, the entry n — 2 has at least 3 entries to its 
right in tt' . By comparing this to the possible types that it may take in Lemma 11.21 
we find that the only possibility is 

tt' G ( *(n - 2)*(n - 4)*(n - 3)1 ). 

This happens if and only if D = 1 and n — 4, n — 3 G C, which gives case 5(h). 

For the two final cases in which n — 2 and n — 1 are to the left of n, it is not 
possible for tt to contain a forbidden pattern of order n — 4, since at least one of 
n — 2, n — 1 and n is needed in order to witness such a pattern, and none of them 
are in the rightmost four positions of the permutation. 

The number of permutations of each type is shown in column three of Table [TJ 
Adding these gives (47n^ — 223n + 240) {n — 4) !/6 which may be be rewritten as in 
formula (EH). □ 



Corollary 2.3. For n > 6, the collection of {n — 4:)-stack sortable permutations in 
&n are those permutations that are not of the types listed in Lemma ll.ll Lemma \l.2\ 
or TableM The number of these is {n - 4)!(3n-* - 18n^ - An'^ + 158n - 192)/3. 

It is now straightforward to write down (a lengthy expression for) the descent 
polynomial of the (n — 4)-stack sortable permutations, that is, the polynomial whose 
k-th coefficient is the number of (n — 4)-stack sortable permutations with exactly 
k descents (a descent in tt = 0102 . . . a„ is an i such that > a^+i). 

The following conjecture is based on computer generated data for n < 13. 

Conjecture 2.4. For all n > 8, the number of permutations in En „_4 is 

<!i^(sM("-)..„»("-)....("-).™3o). 
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or equivalently, the number of {n — 5)-stack sortable permutations is 




- (60n^ - 600n^ + 506n^ + 11241n^ - 38369n + 34236) . 



We end with a conjecture about the form of an expression for the number of 
permutations needing exactly n — k stack sorts. This has been verified for all 
n < 14 and all relevant k. 

Conjecture 2.5. For all n > 2k, the number of permutations in En^n-k mO'V be 
written as 
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fc-i 



where Oj G N, 
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